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The RICIS Concept 


The University of Houston-Clear Lake established the Research Institute for 
Computing and information Systems (RICIS) in 1986 to encourage the NASA 
Johnson Space Center (JSC) and local industry to actively support research 
in the computing and information sciences. As part of this endeavor, UHCL 
proposed a partnership with JSC to jointly define and manage an integrated 
program of research in advanced data processing technology needed for JSC's 
main missions, including administrative, engineering and science responsi- 
bilities. JSC agreed and entered into a continuing cooperative agreement 
with UHCL beginning in May 1986, to jointJy plan and execute such research 
through RICIS. Additionally, under Cooperative Agreement NCC 9-16, 
computing and educational facilities are shared by the twg institutions to 
conduct the research. 

The UHCL/RICIS mission is to conduct, coordinate, and disseminate research 
and professional level education in computing and information systems to 
serve the needs of the government, industry, community and academia. 
RICIS combines resources of UHCL and its gateway affiliates to research and 
develop materials, prototypes and publications on topics of mutual interest 
to its sponsors and researchers. Within UHCL, the mission is being 
implemented through interdisciplinary Involvement of faculty and students 
from each of the four schools: Business and Public Administration, Educa- 
tion, Human Sciences and Humanities, and Natural and Applied Sciences. 
RICIS also collaborates with industry in a companion program. This program 
is focused on serving the research and advanced development needs of 
industry. 

Moreover, UHCL established relationships with other universities and re- 
search organizations, having common research interests, to provide addi- 
tional sources of expertise to conduct needed research. For example, UHCL 
has entered into a special partnership with Texas A&M University to help 
oversee RICIS research and education programs, while other research 
organizations are involved via the "gateway’' concept. 

A major role of RICIS then is to find the best match of sponsors, researchers 
and research objectives to advance knowledge in the computing and informa- 
tion sciences. RICIS, working jointJy with its sponsors, advises on research 
needs, recommends principals for conducting the research, provides tech- 
nical and administrative support to coordinate the research and integrates 
technical results into the goals of UHCL, NASA/ JSC and industry. 
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Preface 


This document summarizes, as part of deliverable #3, the results cf the first winter workshop on 
Gauging Where We Are In The Verification And Validation Of KBSs co-sponsored by NASA/JSC and the 
Electric Power Research Institute (EPRI). The workshop convened at the JSC Gilruth Center over Feb- 
ruary 3 and 4, 1992. 


Introduction 

For the past few years, researchers interested in advancing the state of the practice in verification and 
validation (V&V) of Knowledge-Based systems (KBSs) have met periodically to address significant 
issues related to this goal. This same group met on February 3 and 4, 1992 at JSC's Gilruth Center to 
begin addressing the following issues: 

• Developing guidelines for selection and/or generation of benchmark knowledge-bases 

• Developing criteria for comparing specific V&V tools 

• Developing criteria for more general methods and approaches 

• Defining the scope for tools/methods by examining V&V analysis already performed on existing 
systems 

• Establish what kinds of results are desired based on these comparisons 

A separate sub-group was defined for each of the issues raised above. Each of these groups met 
separately to discuss their specific issue. Results from these separate discusssions were summarized 
and then presented to the rest of the workshop attendees. 

Summary of Results 

The intent of the workshop was to start moving KBS V&V research in the direction of providing tangible 
"products” that a KBS developer can use. These results can be summarized as follows: 

• Provide developers with a classification of kinds of errors versus the kind of KBS being developed. 
In the near term, research will focus on identifying the kinds of experiences encountered during 
KBS development of “real” KBSs. These will be stored in a repository and will serve as the foun- 
dation for the rest of the activities described here. 

• Provide developers with specific criteria for comparing one test tool against another. One specific 
approach to be pursued is “benchmarking." With this approach, a KBS developer can use either 
"canned” KBSs with seeded errors or existing KBSs with known errors to evaluate a given tools 
ability to satisfactorily identify errors. 
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• Provide developers with help in identifying the appropriate artifacts to be produced during KBS 
development. 

* 

Based on these results, specific actions were given/taken to make progress toward these objectives. 

The progress will be reviewed at the AAAI workshop to be held in July of 1992. 

Implications 

This section summarizes, based on the workshop results, some key implications regarding the ES V&V 

Guidelines workshop developed under this contract. 

1. Show how existing trace and diagnostic capabilities of ES shells can be used to help with testing. 
Nick Sizemore made this suggestion when discussin the lack of available testing tools. 

2. Include some information on error categories. There is existing information on error categories for 
conventional software and we could come up with at least some information (e.g., a partial list) for 
expert systems. If we expand on this idea, we could go as far as generating some checklists that 
could be used for inspections and testing. 

3. Include some information on artifacts (i.e., work products) that could be useful in the V&V process 
(e.g., test plans, test results, problem descriptions, user's guides, meta-knowledge, etc.). This 
would have to be done in a way that does not overwhelm the student (i.e., documentation generally 
has a negative impact so describing lots of different documents that they should develop might 
negatively influence their decision to apply V&V approaches). 

4. Include some information on different kinds of expert systems along with information relating types 
of errors to kinds of expert systems 
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